Reduce-or-Retreat Methods in Optimization

We develop a very broad framework of reduce-or-retreat methods for numerical optimization, by identifying the essential components of the many optimization methods that fit this category. We show that a subsequence of the iterates generated by reduce-or-retreat methods approaches stationarity in the sense of having diminishing gradients (or more generally, subgradients). Much of the paper is dedicated to working out the specific details of the general results in particular cases of numerical methods falling in two traditionally distinct categories covered by our framework: Model-based methods (including classical trust-region methods as well as derivative-free methods) and descent methods (including steepest descent, Nelder-Mead, and more general pattern search methods).